<html>
<head><meta charset="utf-8"><title>Experiments on registry index cache · t-cargo · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/index.html">t-cargo</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html">Experiments on registry index cache</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="240743828"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240743828" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240743828">(May 30 2021 at 07:43)</a>:</h4>
<p>I am doing some experiments on improvement of registry index caching . However, the initial result seems not as great as I expect. I would love to see some inputs from you and let's see how far we can go. Perhaps it can prove that the idea is just a strawman so we can postpone it then.</p>
<p><a href="https://github.com/rust-lang/cargo/issues/6908#issuecomment-850940152">https://github.com/rust-lang/cargo/issues/6908#issuecomment-850940152</a></p>



<a name="240762537"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240762537" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240762537">(May 30 2021 at 16:13)</a>:</h4>
<p>This is great, thanks for working on it!  We've been discussing using sqlite or something ilke it for a while now, but I haven't had a chance to actually test it, so this is awesome!  I've also been thinking about using it for fingerprints, and particularly for "last-used" data where it captures a timestamp for the last time each dependency was used (which can be later used for garbage collection).</p>
<p>Testing on <code>cargo</code> may not be a very good stress test, since its dependency tree isn't large.  Have you had a chance to test it on a larger project?  Some that I can think of:</p>
<ul>
<li>cargo: 130</li>
<li>rust: 518</li>
<li>tikv: 552</li>
<li>firefox: 577</li>
<li>diem: 653</li>
<li>servo: 658</li>
<li>paritytech/substrate: 896</li>
</ul>



<a name="240762662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240762662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240762662">(May 30 2021 at 16:15)</a>:</h4>
<p>Also, <code>generate-lockfile</code> may not be a good test since it accesses the network.  I think <code>generate-lockfile -Zno-index-update</code> should work better.</p>



<a name="240782765"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240782765" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240782765">(May 31 2021 at 01:16)</a>:</h4>
<p>Here is the result. Seems no significant difference between SQLite and filesystem way.</p>
<p><a href="https://github.com/rust-lang/cargo/issues/6908#issuecomment-851105061">https://github.com/rust-lang/cargo/issues/6908#issuecomment-851105061</a></p>



<a name="240783359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240783359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240783359">(May 31 2021 at 01:35)</a>:</h4>
<p>mostly just here as a lurker, but i'm one of the maintainers of rusqlite and i notice a few small performance things you're doing wrong.</p>
<p>i suspect they wouldn't make a dent here, but how hard would it be for you to check if using WITHOUT ROWID on the table (<a href="https://sqlite.org/withoutrowid.html">https://sqlite.org/withoutrowid.html</a> — given your schema it seems likely to be a win) and preparing the statements with prepare_cached (<a href="https://docs.rs/rusqlite/0.25.3/rusqlite/struct.Connection.html#method.prepare_cached">https://docs.rs/rusqlite/0.25.3/rusqlite/struct.Connection.html#method.prepare_cached</a>) helps</p>



<a name="240783409"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240783409" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240783409">(May 31 2021 at 01:36)</a>:</h4>
<p>also enabling WAL mode <a href="https://sqlite.org/wal.html">https://sqlite.org/wal.html</a></p>



<a name="240784637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240784637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240784637">(May 31 2021 at 02:10)</a>:</h4>
<p>I remembered I've tried <code>WITHOUT ROWID</code> but no significant difference. I've also done experiments for external blob but in reality most packages won't be <a href="https://sqlite.org/intern-v-extern-blob.html">larger than 100KB</a>, so it won't get much improvement as well.</p>
<p>Anyway, all are valuable advices. Thank you <span class="user-mention" data-user-id="209168">@Thom Chiovoloni</span>!  I'll try these and update the result once done. <span aria-label="blush" class="emoji emoji-1f60a" role="img" title="blush">:blush:</span></p>



<a name="240786950"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240786950" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240786950">(May 31 2021 at 03:14)</a>:</h4>
<p>The new benchmark result for SQLite is out. Please tell me if I did something wrong.</p>
<p><a href="https://github.com/rust-lang/cargo/issues/6908#issuecomment-851136439">https://github.com/rust-lang/cargo/issues/6908#issuecomment-851136439</a></p>



<a name="240789877"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240789877" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240789877">(May 31 2021 at 04:20)</a>:</h4>
<p>Huh, interesting. yeah that looks basically like what I had imagined</p>



<a name="240790773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240790773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240790773">(May 31 2021 at 04:35)</a>:</h4>
<p>I might poke this a bit tomorrow (or maybe the next day — holiday and all) to see if different combinations of those make a difference, but yeah i expect you won't get many more wins here.</p>



<a name="240814057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240814057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240814057">(May 31 2021 at 09:58)</a>:</h4>
<p>Isn't the entire proposition that the performance wins would be mostly on Windows which has a pretty slow disk I/O when presented with many files at once? UNIX systems are pretty good at handling many small files.</p>



<a name="240850369"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240850369" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240850369">(May 31 2021 at 16:12)</a>:</h4>
<p>oh, i thought thats where the tests were run</p>



<a name="240850527"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240850527" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240850527">(May 31 2021 at 16:14)</a>:</h4>
<p>its probably worth ensuring windows defender is on too, since IME it hurts that kind of disk IO a lot too, and i think there's a good fraction of users that would have it on (if not most)</p>



<a name="240884656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240884656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240884656">(Jun 01 2021 at 01:09)</a>:</h4>
<p>Sorry I cannot access my Windows machine at this moment, but at least we know that SQLite won't make it worse on macOS <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>



<a name="240953462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240953462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240953462">(Jun 01 2021 at 14:51)</a>:</h4>
<p>I have access to Win computers. Although I have moved on from the laptop with the failing hard drive and aggressive antivirus that I was using when I suggested the format change. But I can run tests on some Win/SSD computers in the next few days, if that would be helpful. How are we running tests?</p>



<a name="240975646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240975646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240975646">(Jun 01 2021 at 17:25)</a>:</h4>
<p><span class="user-mention silent" data-user-id="120518">Eric Huss</span> <a href="#narrow/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache/near/240762537">said</a>:</p>
<blockquote>
<ul>
<li>cargo: 130</li>
<li>rust: 518</li>
<li>tikv: 552</li>
<li>firefox: 577</li>
<li>diem: 653</li>
<li>servo: 658</li>
<li>paritytech/substrate: 896</li>
</ul>
</blockquote>
<ol>
<li>Clone the aforementioned repositories</li>
<li><code>cargo fetch</code> for each project</li>
<li>Run the script mentioned in this <a href="https://github.com/rust-lang/cargo/issues/6908#issuecomment-851105061">comment</a>. </li>
</ol>
<p>The folder structure looks like below:</p>
<div class="codehilite"><pre><span></span><code>benchmark/
├── diem/
├── gecko-dev/
├── rust/
├── servo/
├── substrate/
├── tikv/
└── bench.bash*
</code></pre></div>
<p>Note that SQLite branch won't auto-create <code>index/.cache</code> folder so we need manually create it (by passing <code>-p</code> flag to <code>hyperfine</code>).</p>
<p>Also, I <em>may</em> have access to PC in the weekend. If you don't mind we can wait.</p>



<a name="240976317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240976317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240976317">(Jun 01 2021 at 17:30)</a>:</h4>
<p>Am I right in my interpretation that so far the numbers collected aren't showing any clear winner on any platform? (e.g. everything is within a few percent of each other)</p>



<a name="240976416"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240976416" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240976416">(Jun 01 2021 at 17:31)</a>:</h4>
<p>(oh also to preface this is some awesome work, thanks for driving this <span class="user-mention" data-user-id="245964">@Weihang Lo</span> !)</p>



<a name="240976561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240976561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240976561">(Jun 01 2021 at 17:32)</a>:</h4>
<p>if my interpretation is right, though, some good next steps might be to break out the profilers and see where time is being spent. There may still be stray file accesses here and there or something like that. I know that <code>.fingerprint</code> stuff for the build directory is also likely huge in terms of number of files accessed and it may benefit from being accelerated too.</p>



<a name="240977548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/240977548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#240977548">(Jun 01 2021 at 17:39)</a>:</h4>
<p><span class="user-mention" data-user-id="245964">@Weihang Lo</span> If you are willing to continue pushing this forward, I am happy to wait until it's it convenient for you!<br>
<span class="user-mention" data-user-id="116015">@Alex Crichton</span> My understanding is that we have run all tests on OS where the current approach is expected to be fast, so braking even is a good sign.</p>



<a name="241150087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241150087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241150087">(Jun 02 2021 at 13:41)</a>:</h4>
<p>Finished the experiment on Windows. There are some performance gains for SQLite on Windows. Detail are listed in the issue comment.<br>
<a href="https://github.com/rust-lang/cargo/issues/6908#issuecomment-853034677">https://github.com/rust-lang/cargo/issues/6908#issuecomment-853034677</a></p>



<a name="241163304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241163304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241163304">(Jun 02 2021 at 15:11)</a>:</h4>
<p>that's great! i hesitate to ask since i'd love for y'all to be able to use it, but would it cause a licensing problem?</p>
<p>that is to say, rusqlite is MIT, not dual Apache2/MIT like cargo (and unfortunately rusqlite is an old project with many contributors over time — i'm unsure about the plausibility of being able to get the sign-off to switch to dual mit/apache2)</p>



<a name="241163688"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241163688" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241163688">(Jun 02 2021 at 15:14)</a>:</h4>
<p>I don't think that is a problem but don't know enough to justify why it is ok. <span class="user-mention" data-user-id="239881">@Josh Triplett</span> probably can give an authoritative answer.</p>



<a name="241164285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241164285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241164285">(Jun 02 2021 at 15:18)</a>:</h4>
<p>i just realized i could check if you already have non-Apache2 crates, and according to cargo-license, these existing crates in cargo's dep tree are MIT (and not MIT/Apache).</p>
<blockquote>
<p>ansi_term, atty, build_const, clap, commoncrypto, commoncrypto-sys, crypto-hash, curl, curl-sys, fwdansi, matches, miniz_oxide, miniz_oxide_c_api, openssl-sys, redox_syscall, redox_syscall, redox_termios, schannel, strsim, termion, textwrap</p>
</blockquote>
<p>obviously not a guarantee (it could be a problem that it's using these now) but seems like a good indicator that i was being overly paranoid</p>



<a name="241169155"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241169155" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241169155">(Jun 02 2021 at 15:52)</a>:</h4>
<p>I think the improvement is enough for the next step, but not sure what to do. Should we continue the work on SQLite integration and hide the feature behind a flag? Can someone helps benchmark on Linux and more other platforms/machines? Do we need a RFC? Is there any possible solution worth a try?</p>



<a name="241170129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241170129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241170129">(Jun 02 2021 at 15:59)</a>:</h4>
<p>This is an internal detail and so will not need an RFC.<br>
I would like to move most/all of our "on disc" databases to one self describing system. So I am in fave of pushing on a SQLite solution, but it is up to the full team.</p>



<a name="241175381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241175381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241175381">(Jun 02 2021 at 16:38)</a>:</h4>
<p><span class="user-mention silent" data-user-id="120179">Eh2406</span> <a href="#narrow/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache/near/241163688">said</a>:</p>
<blockquote>
<p>I don't think that is a problem but don't know enough to justify why it is ok. <span class="user-mention silent" data-user-id="239881">Josh Triplett</span> probably can give an authoritative answer.</p>
</blockquote>
<p>An MIT-licensed dependency is absolutely not an issue. It's a permissive license that's compatible with everything.</p>
<p>Rust uses the <code>MIT OR Apache-2.0</code> dual-license because of the Apache license's patent grant, to protect the Rust project from the possibility of someone submitting code to Rust and subsequently making a patent claim about that code. We want that to be the case for code contributed to the Rust project itself. We don't need every third-party library we use to have the same patent grant, particularly those that many other projects also use.</p>



<a name="241339945"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241339945" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241339945">(Jun 03 2021 at 01:34)</a>:</h4>
<p>It is great to have a discussion about possible extensions SQLite will bring for Cargo, so we can tell whether it is worthwhile. Here listed what I think of at this moment:</p>
<ul>
<li>Timestamp for fingerprint <span class="user-mention" data-user-id="120518">@Eric Huss</span> mentioned.</li>
<li>The performance improvement on Windows so for in registry index cache.</li>
<li>Large blobs such as <code>.cargo/registry/cache</code> <a href="https://www.sqlite.org/fasterthanfs.html">may benefit from SQLite</a> but also need experiments.</li>
<li>SQLite has different <a href="https://sqlite.org/lockingv3.html">lock-mode</a> and <a href="https://sqlite.org/wal.html">WAL</a> so there is a chance to reduce some flock usage. This seems neutral to me just FYI.</li>
</ul>



<a name="241340429"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241340429" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241340429">(Jun 03 2021 at 01:45)</a>:</h4>
<p><span class="user-mention" data-user-id="245964">@Weihang Lo</span> File locking is one of the issues I investigated in the past with SQLite, but I was still uncertain about how it works.  From what I understand from <a href="https://sqlite.org/howtocorrupt.html">https://sqlite.org/howtocorrupt.html</a>, if the filesystem doesn't support locking, there is a high likelyhood it can corrupt the database.  That seems like it could be a very bad failure mode.  Cargo's current behavior isn't great either, but I think it can still chug along.  We made need to reconsider cargo's behavior, I'm not sure.</p>



<a name="241341214"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241341214" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241341214">(Jun 03 2021 at 02:01)</a>:</h4>
<p>Comparing to Cargo's current behavior, is it similar to a choice between "corrupt some files" v.s. "corrupt the whole database"?</p>



<a name="241341736"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241341736" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eric Huss <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241341736">(Jun 03 2021 at 02:12)</a>:</h4>
<p>To be honest, I'm not really sure what happens when there is no locking.  I think the git index should be relatively safe.  I'm not sure what might happen with the index cache, but cache errors are ignored and it falls back to the git source of truth.  I'm not sure about other parts like fingerprints and rustc artifacts and linking.</p>
<p>In general I'm not sure if it is a big problem.  rustc's incremental mode enforces that locking is enabled, so in practice most people doing daily development will figure out that they need to use a different filesystem.  Other scenarios, like CI, are unlikely to have concurrent processes running.</p>
<p>One idea is to enforce locking, and if it is not supported, require the user to take some action to acknowledge they will not run concurrent processes (like an env var or config setting), but I feel like that is not a good idea.  So, I'm not sure what are good options.</p>



<a name="241346613"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346613" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346613">(Jun 03 2021 at 04:09)</a>:</h4>
<p>see also <a href="https://www.sqlite.org/atomiccommit.html#_broken_locking_implementations">https://www.sqlite.org/atomiccommit.html#_broken_locking_implementations</a> which goes into a bit more detail about the problem cases</p>



<a name="241346754"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346754" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346754">(Jun 03 2021 at 04:13)</a>:</h4>
<p>it's worth noting that it indicates extent of the bugs known for NFS is</p>
<blockquote>
<p>We have received reports of implementations of both Windows network filesystems and NFS in which locking was subtly broken. We can not verify these reports, but as locking is difficult to get right on a network filesystem we have no reason to doubt them</p>
</blockquote>
<p>which feels likely to not cause many problems in practice</p>



<a name="241346807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346807">(Jun 03 2021 at 04:14)</a>:</h4>
<p>that said, from working on firefox places (bookmarks, history) in the past, one thing i can say concretely is that db corruption is a real thing that happens to some small portion of users.</p>



<a name="241346812"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346812" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346812">(Jun 03 2021 at 04:14)</a>:</h4>
<p>i've always attributed it to on-disc storage errors</p>



<a name="241346828"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346828" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346828">(Jun 03 2021 at 04:15)</a>:</h4>
<p>For index cache, is there any chance, say, do a <a href="https://www.sqlite.org/pragma.html#pragma_quick_check">check</a> when opening the database, and if corruption happened just remove the file and start caching from scratch?</p>
<p>Though there is a overhead on integrity check.</p>



<a name="241346840"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346840" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346840">(Jun 03 2021 at 04:15)</a>:</h4>
<p>there's also <a href="https://sqlite.org/cksumvfs.html">https://sqlite.org/cksumvfs.html</a> (rusqlite doesn't support this but could without lots of work)</p>



<a name="241346892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346892">(Jun 03 2021 at 04:16)</a>:</h4>
<p>i don't know the performance overhead of this. in practice it seems likely that this sort of corruption is likely to be present on flat files just as much as a db, though</p>



<a name="241346897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346897">(Jun 03 2021 at 04:17)</a>:</h4>
<p><span class="user-mention" data-user-id="209168">@Thom Chiovoloni</span> NFS has been an known issue for a long time <span aria-label="joy" class="emoji emoji-1f602" role="img" title="joy">:joy:</span><br>
<a href="https://github.com/rust-lang/cargo/blob/163097cd/src/cargo/util/flock.rs#L284-L296">https://github.com/rust-lang/cargo/blob/163097cd/src/cargo/util/flock.rs#L284-L296</a></p>



<a name="241346972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241346972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241346972">(Jun 03 2021 at 04:18)</a>:</h4>
<p>hm, i see</p>



<a name="241347041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Experiments%20on%20registry%20index%20cache/near/241347041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Weihang Lo <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Experiments.20on.20registry.20index.20cache.html#241347041">(Jun 03 2021 at 04:21)</a>:</h4>
<blockquote>
<p>this sort of corruption is likely to be present on flat files just as much as a db</p>
</blockquote>
<p>Yeah. That's why I see it as a "corrupt some v.s. corrupt all" problem. For cache we can just ditch all corrupted files but for artifacts and fingerprints it may be more painful.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>